<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" >

<title>使用TensorRT对模型进行压缩加速 | KALUO</title>
<meta name="description" content="温故而知新">

<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">

<link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.7.2/css/all.css" integrity="sha384-fnmOCqbTlWIlj8LyTjo7mOUStjsKC4pOpQbqyi7RrhN7udi9RwhKkMHpvLbHG9Sr" crossorigin="anonymous">
<link rel="shortcut icon" href="https://kaluo-zz.github.io/favicon.ico?v=1591084625855">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.10.0/katex.min.css">
<link rel="stylesheet" href="https://kaluo-zz.github.io/styles/main.css">



<script src="https://cdn.jsdelivr.net/npm/vue/dist/vue.js"></script>
<script src="https://cdn.bootcss.com/highlight.js/9.12.0/highlight.min.js"></script>

<link rel="stylesheet" href="https://unpkg.com/aos@next/dist/aos.css" />



  </head>
  <body>
    <div id="app" class="main">

      <div class="sidebar" :class="{ 'full-height': menuVisible }">
  <div class="top-container" data-aos="fade-right">
    <div class="top-header-container">
      <a class="site-title-container" href="https://kaluo-zz.github.io">
        <img src="https://kaluo-zz.github.io/images/avatar.png?v=1591084625855" class="site-logo">
        <h1 class="site-title">KALUO</h1>
      </a>
      <div class="menu-btn" @click="menuVisible = !menuVisible">
        <div class="line"></div>
      </div>
    </div>
    <div>
      
        
          <a href="/" class="site-nav">
            首页
          </a>
        
      
        
          <a href="/archives" class="site-nav">
            归档
          </a>
        
      
        
          <a href="/tags" class="site-nav">
            标签
          </a>
        
      
        
          <a href="/post/about" class="site-nav">
            关于
          </a>
        
      
    </div>
  </div>
  <div class="bottom-container" data-aos="flip-up" data-aos-offset="0">
    <div class="social-container">
      
        
      
        
      
        
      
        
      
        
      
    </div>
    <div class="site-description">
      温故而知新
    </div>
    <div class="site-footer">
      Powered by <a href="https://github.com/getgridea/gridea" target="_blank">Gridea</a> | <a class="rss" href="https://kaluo-zz.github.io/atom.xml" target="_blank">RSS</a>
    </div>
  </div>
</div>


      <div class="main-container">
        <div class="content-container" data-aos="fade-up">
          <div class="post-detail">
            <h2 class="post-title">使用TensorRT对模型进行压缩加速</h2>
            <div class="post-date">2019-07-01</div>
            
            <div class="post-content">
              <h3 id="安装tensorrt5122">安装TensorRT5.1.2.2</h3>
<ol>
<li>
<p>到NVIDIA<a href="https://developer.nvidia.com/tensorrt">官网</a>下载对应Ubuntu系统的TensorRT5.1.2.2。</p>
</li>
<li>
<p>新建目录TensorRT，将下载的文件移动至该目录下，并解压：</p>
<pre><code class="language-bash">username@hostname:~/TensorRT$ tar xzvf TensorRT-5.1.2.2.Ubuntu-16.04.4.x86_64-gnu.cuda-9.0.cudnn7.5.tar.gz
</code></pre>
</li>
<li>
<p>添加TensorRT的绝对路径<code>LIB</code>目录到环境变量<code>LD_LIBRARY_PATH</code>:</p>
<pre><code class="language-bash">username@hostname:~/TensorRT/TensorRT-5.1.2.2$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/kaluo/TensorRT/TensorRT-5.1.2.25/lib
</code></pre>
</li>
<li>
<p>安装Python TensorRT wheel文件</p>
<pre><code class="language-bash">username@hostname:~/TensorRT/TensorRT-5.1.2.2$ cd python/
username@hostname:~/TensorRT/TensorRT-5.1.2.2/python$ ls
tensorrt-5.1.2.2-cp27-none-linux_x86_64.whl
tensorrt-5.1.2.2-cp34-none-linux_x86_64.whl
tensorrt-5.1.2.2-cp35-none-linux_x86_64.whl
tensorrt-5.1.2.2-cp36-none-linux_x86_64.whl
tensorrt-5.1.2.2-cp37-none-linux_x86_64.whl
username@hostname:~/TensorRT/TensorRT-5.1.2.2/python$ pip install tensorrt-5.1.2.2-cp27-none-linux_x86_64.whl #这里使用python2.7
Processing ./tensorrt-5.1.2.2-cp27-none-linux_x86_64.whl
Installing collected packages: tensorrt
Successfully installed tensorrt-5.1.2.2
</code></pre>
</li>
<li>
<p>如果需要将TensorRT和TensorFlow一起使用，则需要安装python UFF wheel文件。</p>
<pre><code class="language-bash">username@hostname:~/TensorRT/TensorRT-5.1.2.2$ cd uff/
username@hostname:~/TensorRT/TensorRT-5.1.2.2/uff$ ls
uff-0.6.3-py2.py3-none-any.whl
username@hostname:~/TensorRT/TensorRT-5.1.2.2/uff$ pip install uff-0.6.3-py2.py3-none-any.whl
</code></pre>
</li>
<li>
<p>安装python <code>graphsurgeon</code> wheel文件。</p>
<pre><code class="language-bash">username@hostname:~/TensorRT/TensorRT-5.1.2.2/graphsurgeon$ pip install graphsurgeon-0.4.0-py2.py3-none-any.whl
Processing ./graphsurgeon-0.4.0-py2.py3-none-any.whl
Installing collected packages: graphsurgeon
Successfully installed graphsurgeon-0.4.0
</code></pre>
</li>
</ol>
<blockquote>
<h3 id="43-tar-file-installation"><a href="https://docs.nvidia.com/deeplearning/sdk/tensorrt-install-guide/index.html#installing-tar">4.3. Tar File Installation</a></h3>
<p>Note: Before issuing the following commands, you'll need to replace 5.1.x.x with your specific TensorRT version. The following commands are examples.</p>
<ol>
<li>
<p>Install the following dependencies, if not already present:</p>
<ul>
<li>Install the CUDA Toolkit 9.0, 10.0 or 10.1</li>
<li>cuDNN 7.5.0</li>
<li>Python 2 or Python 3 (Optional)</li>
</ul>
</li>
<li>
<p><a href="https://docs.nvidia.com/deeplearning/sdk/tensorrt-install-guide/index.html#downloading">Download</a> the TensorRT tar file that matches the Linux distribution you are using.</p>
</li>
<li>
<p>Choose where you want to install TensorRT. This tar file will install everything into a subdirectory called TensorRT-5.1.x.x.</p>
</li>
<li>
<p>Unpack the tar file.</p>
<pre><code class="language-bash">$ tar xzvf TensorRT-5.1.x.x.Ubuntu-1x.04.x.x86_64-gnu.cuda-x.x.cudnn7.x.tar.gz
</code></pre>
<p>Where:</p>
<ul>
<li>5.1.x.x is your TensorRT version</li>
<li>Ubuntu-1x.04.x is 14.04.5, 16.04.4 or 18.04.1</li>
<li>cuda-x.x is CUDA version 9.0, 10.0, or 10.1</li>
<li>cudnn7.x is cuDNN version 7.5</li>
</ul>
<p>This directory will have sub-directories like <code>lib</code>, <code>include</code>,<code>data</code>, etc…</p>
<pre><code class="language-bash">$ ls TensorRT-5.1.x.x
bin  data  doc  graphsurgeon  include  lib  python  samples  targets  TensorRT-Release-Notes.pdf  uff
</code></pre>
</li>
<li>
<p>Add the absolute path to the TensorRT lib directory to the environment variable LD_LIBRARY_PATH:</p>
<pre><code class="language-bash">$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:&lt;eg:TensorRT-5.1.x.x/lib&gt;
</code></pre>
</li>
<li>
<p>Install the Python TensorRT wheel file.</p>
<pre><code class="language-bash">$ cd TensorRT-5.1.x.x/python
</code></pre>
<pre><code class="language-bash">$ sudo pip2 install tensorrt-5.1.x.x-cp27-none-linux_x86_64.whl
</code></pre>
<pre><code class="language-bash">$ sudo pip3 install tensorrt-5.1.x.x-cp3x-none-linux_x86_64.whl
</code></pre>
</li>
<li>
<p>Install the Python UFF wheel file. This is only required if you plan to use TensorRT with TensorFlow.</p>
<pre><code class="language-bash">$ cd TensorRT-5.1.x.x/uff
</code></pre>
<pre><code class="language-bash">$ sudo pip2 install uff-0.6.3-py2.py3-none-any.whl
</code></pre>
<pre><code class="language-bash">$ sudo pip3 install uff-0.6.3-py2.py3-none-any.whl
</code></pre>
<pre><code class="language-bash">$ which convert-to-uff
/usr/local/bin/convert-to-uff
</code></pre>
</li>
<li>
<p>Install the Python graphsurgeon wheel file.</p>
<pre><code class="language-bash">$ cd TensorRT-5.1.x.x/graphsurgeon
</code></pre>
<pre><code class="language-bash">$ sudo pip2 install graphsurgeon-0.4.0-py2.py3-none-any.whl
</code></pre>
<pre><code class="language-bash">$ sudo pip3 install graphsurgeon-0.4.0-py2.py3-none-any.whl
</code></pre>
</li>
<li>
<p>Verify the installation:</p>
<ol>
<li>Ensure that the installed files are located in the correct directories. For example, run the tree -d command to check whether all supported installed files are in place in the lib, include, data, etc… directories.</li>
<li>Build and run one of the shipped samples, for example, sampleMNIST in the installed directory. You should be able to compile and execute the sample without additional settings. For more information about sampleMNSIT, see the <a href="https://docs.nvidia.com/deeplearning/sdk/tensorrt-sample-support-guide/index.html#mnist_sample">TensorRT Sample Support Guide</a>.</li>
<li>The Python samples are in the samples/python directory.</li>
</ol>
</li>
</ol>
</blockquote>
<h1 id="pip-install-error-with-pycuda">Pip install error with PyCUDA</h1>
<h2 id="problem">Problem</h2>
<ul>
<li>
<p>I tried to install PyCUDA using <code>pip</code>:</p>
</li>
<li>
<p>The installation tries to compile a few C++ files and it failed on the very first file with this error:</p>
<pre><code class="language-bash">In file included from src/cpp/cuda.cpp:1:0:
src/cpp/cuda.hpp:14:18: fatal error: cuda.h: No such file or directory
#include &lt;cuda.h&gt;
                ^
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
</code></pre>
</li>
</ul>
<h2 id="investigation">Investigation</h2>
<p>This error was strange because I had set <code>CUDA_ROOT</code> and had added the <code>bin</code> path of CUDA installation to <code>PATH</code> environment variable. So, the installer should have found <code>cuda.h</code> which I could see was present in <code>$CUDA_ROOT/include</code></p>
<p>To see what was happening, I tried the same command with verbosity:</p>
<pre><code class="language-bash">$ sudo pip -vvv install pycuda
</code></pre>
<ul>
<li>Now I could see that it was failing to find <code>nvcc</code>.</li>
<li>On downloading the source code of PyCUDA and checking <code>setup.py</code>, I saw that the check for <code>nvcc</code> was used to figure out the <code>CUDA_ROOT</code> and <code>CUDA_INC_DIR</code>.</li>
<li>The reason <code>nvcc</code> was not visible was that <code>CUDA_ROOT</code> was set for my user, but this <code>PATH</code> is not visible when a command is run under <code>sudo</code>, as described here. The solution was to make the CUDA <code>bin</code> path visible to <code>sudo</code>.</li>
</ul>
<h2 id="solution">Solution</h2>
<p>To make the <code>$CUDA_ROOT/bin</code> available in <code>PATH</code> for <code>sudo</code>, we can follow the steps described here. For example, on my system with CUDA 7.0 I followed these steps:</p>
<ul>
<li>
<p>Created a new file <code>/etc/profile.d/cuda.sh</code> and added this line:</p>
<pre><code class="language-bash">export PATH=/usr/local/cuda-7.0/bin:$PATH
</code></pre>
</li>
<li>
<p>Opened <code>root</code> shell without resetting <code>PATH</code> and ran the pip installation:</p>
</li>
</ul>
<pre><code class="language-bash">$ sudo su -
$ pip install pycuda
</code></pre>
<p>This worked and PyCUDA was installed successfully!</p>

            </div>
            
              <div class="tag-container">
                
                  <a href="https://kaluo-zz.github.io/tag/tMhk8q8aw" class="tag">
                    TensorRT
                  </a>
                
                  <a href="https://kaluo-zz.github.io/tag/Pd55hi5zg" class="tag">
                    模型压缩
                  </a>
                
              </div>
            
            
              <div class="next-post">
                <div class="next">下一篇</div>
                <a href="https://kaluo-zz.github.io/post/pytorch-huo-qu-mo-xing-xin-xi-bao-gua-flopscan-shu-liang">
                  <h3 class="post-title">
                    pytorch获取模型信息（包括FLOPs、参数量）
                  </h3>
                </a>
              </div>
            

            

          </div>

        </div>
      </div>
    </div>

    <script src="https://unpkg.com/aos@next/dist/aos.js"></script>

<script type="application/javascript">

AOS.init();

hljs.initHighlightingOnLoad()

var app = new Vue({
  el: '#app',
  data: {
    menuVisible: false,
  },
})

</script>




  </body>
</html>
